Business Analyst Community & Resources | Modern Analyst

Measuring the Success of Training Activities

Jarett Hailes — Sun, 10 Jan 2010 17:14:00 GMT

I am in charge of a relatively big training effort for a project (approximately 45 live training sessions in 10 weeks, as well as online training opportunities) to assist with the deployment of a new piece of software. The live training alone will involve over 450 people and will be quite in depth and hands on. Training often plays a critical role in the successful adoption of a new product. Without the proper knowledge of how to effectively use a system, clients can become frustrated, ambivalent or hostile to the change that’s occurring, which can ultimately affect the success of the change.

As a result effective training methods are essential to ensure the ultimate success of the product and the project from which the product was spawned. But how can we determine whether the training was useful, relevant, and appropriate for the given audience? Like any other part of a project or initiative, we’d like to have some accountability on the quality of work product. As a result it is important to come up with ways of measuring how successful your training activities are.

While surveys of trainees and the like can help measure the perceived quality of the training itself, on its own such information does not provide us with a true measure of the success of the training process. You need to map the output of the training to its end goals and from there develop measures that will provide an additional level of understanding as to the quality of the training.

Establish Your Outcomes

First, you need to establish your target outcomes, the end results that you foresee as a result of the training activities. Typically I start with qualitative outcome descriptions. For my project, we want to ensure that clients understand how to use the software well and that they can operate the software independently (i.e. no need to ask for help). We also want to ensure that clients realize how they can use the information available to them through the system to resolve data conflicts with each other rather than having to go through a third party. Lastly, we want to reduce the need for a third party monitoring group to have to contact clients and enforce data resolution and processing activities.

Define Your Measures

Once you have your outcomes, you need to develop ways to measure these outcomes. These must be quantifiable results that you can use to accurately demonstrate whether an outcome was met or not. For example, we could use the number of calls and e-mails to the Help desk regarding the product as a measure for how well clients understand the software once the training is deployed. We can also measure the number of times the third party is contacted to resolve a data issue between clients who work in different organizations. Lastly we can count the number of times the third party must contact clients in order for data conflicts to be resolved.

If the training is related to an existing set of processes, you would ideally use measures for which you already have (or can gather prior to) the implementation of the product/solution. This way you can have data not only post-implementation but also pre-implementation and analyze the impact of the product and the training.

Create Goals for Each Measure

Once your measures are in place, come up with target goals for each measure. If you have pre-deployment data, your goals can be about trending rather than absolute figures (for instance, you could set a goal for a 20% reduction in Help desk calls regarding process X instead of saying your goal is to have less than 200 calls regarding process X).

These goals should be evaluated by the relevant stakeholders, approved by the project sponsor and become part of the project’s success criteria.

Some Considerations

If you’re implementing a product that is designed to improve an existing process you probably have some measures and targets for the productivity of the process itself once the product is deployed (e.g. increase order taking throughput per hour by 50%). For these measures, training is only one of several factors going into the end result and cannot be easily removed to analyze the success of training on its own.

However, you could perform some basic ‘A/B’ tests with a limited subset of clients if you wish to determine how much of an impact training has on the product’s overall success. Train one group of clients as per usual and then train another group with either a different set of training materials, style, etc. or not at all. Assuming you control for other relevant factors that may impact the end result (e.g. level of experience of clients, demographics of clients, etc.) this can be an extremely useful to determine how important training was to the overall success of the project.

If not all of your clients have access to the same type or level of training try and segregate their data so you don’t make erroneous conclusions about the success of your training activities.

Deploying a new product, process, or system into the organization is only as effective as the individuals who implement the change. Making sure that training aligns with the end objectives and can be measured to determine its utility is an important aspect of the overall project’s outcomes.

Jarett Hailes
Larimar Consulting Inc.
http://www.larimarconsulting.com

Why Business Analysts Should Not Perform Quality Assurance

Jarett Hailes — Wed, 06 Jan 2010 02:33:00 GMT

I seem to find quite a few job postings for Business Analysts that have a couple of lines in the ‘tasks to perform’ description that scare me like:

Define test strategy and test plan
Develop and execute test cases
Perform stress and load test scenarios

It’s not that I don’t believe I am capable of these tasks at some level of competence; after all most BAs have had to perform these activities at some point in their career. I don’t believe a BA should be expected to do Quality Assurance tasks. I can understand the temptation to leverage Business Analysts to perform Quality Assurance tasks, namely:

BAs already have a good understanding of requirements, use cases, etc. so it should be easy for them to come up with testing approaches and test cases
BAs are ‘analytical’ in nature and thus are well suited to QA activities
After the requirements are done BAs just typically sit around or are moved to another project, so we might as well give them something to do so they can be with the project for its entire lifecycle (and be held accountable for ensuring in the end user needs are met)

While these arguments all have some level of credibility, I don’t believe that having BAs perform QA tasks should be considered good practice. Don’t get me wrong, I believe that BAs need to work closely with testers to ensure that requirements are well understood, that requirements can be traced end-to-end, and to help facilitate discussion amongst the team to develop a testing approach. I don’t even mind having to execute test cases if the team’s in a crunch and need some more people to run through scenarios. But overall, I believe that having a Business Analyst being in charge of testing activities (especially if it’s on the same project as doing BA tasks) is not a good idea for the reasons I discuss below.

Quality Assurance is a Separate Role

As the BABOK V2.0 has so eloquently stated, Business Analysis is about working with stakeholders within an organization to identify solutions that will help the organization meet its goals. The BABOK barely mentions testing, only to indicate that a Tester is “response for determining how to verify that the solution meets the requirements defined by the Business Analyst.” These are two distinct roles that while they involve leveraging common knowledge and artefacts (e.g. requirements) they definitively involve disparate activities, techniques, and tools.

Quality Assurance is its Own Profession

Not only is QA its own role, there are organizations that promote the tasks performed by testers as its own profession. There are even separate bodies of knowledge, designations and certifications that have been developed around QA activities. Like any profession it takes time to become adept at performing QA tasks, just as it does to become proficient at BA tasks.

While individuals may develop expertise in both Business Analysis and Quality Assurance over several years (in which case they can perform either role) it is unlikely that most BAs have the level of knowledge to adequately perform QA tasks.

Bias Inherent in Performing Both Roles

For me this is the most important consideration – if you are the Business Analyst (or one of the BAs) involved in the gathering of requirements, you end up developing certain biases based on your knowledge of the business and the problem(s) that are to be solved by the solution being developed. Such biases are rarely at a conscious level, but do occur frequently.

The BA can become so intimate with the requirements that they may not realize when certain assumptions are undocumented, or may unwittingly ignore certain paths or scenarios that need to be tested because they understand the recourses that can occur in those circumstances. In essence the BA may become so ‘in tune’ with the requirements as documented that they may omit or neglect circumstances or exceptions in testing plans that need to be considered. An unbiased third party without such intimate knowledge would have a much easier time to spot any inconsistencies.

Such biases can impact the veracity of testing activities. For example, if the BA starts to write test cases based on what’s in their head rather than what is documented, traceability can become an issue. If the BA omits certain test case scenarios since they know that they are unlikely to occur, are not documented, or the BA already knows how the system will handle the scenario (since they saw it at the last demo), then not only is the current testing phase impacted, regression testing on future changes to the system can become compromised.

Lost Opportunity to Have Another Set of Eyes in the Process

As noted above, a BA can form a biased view of the solution and the project based on their work and job duties. It never hurts to have another brain in the fold that is tasked with, among other things, finding holes in documented processes, looking for requirements that may be under-developed, and generally ensuring that the solution is free from issues that would cause stakeholders to reject it. There are few downsides to including another person given the risks inherent with one person performing both sets of duties.

Get a Tester

Overall, if your organization has the capacity to have dedicated Quality Assurance professionals they will pay for themselves in spades (just as Business Analysts do). Using BAs for testing purposes may seem like a slick thing to do, but in my opinion it is almost never the right choice given the option to go with a separate individual.

What do you think? Leave a comment and let me know your experiences. Have you been a BA who has been forced into QA duties? Do you think there’s no issue with someone doing ‘double duty’?